Add elasticgraph-proto_ingestion schema artifacts by jwils · Pull Request #1080 · block/elasticgraph

jwils · 2026-03-22T20:52:45Z

Note

Third of a three-PR sequence: #1296 (artifact comment prefixes) and #1297 (gem scaffold) split the supporting changes out, so this PR's diff is now contained entirely to the elasticgraph-proto_ingestion directory.

Why

continue the pluggable ingestion serializer proposal after pulling JSON Schema into its own gem
revive the earlier protobuf prototype from Add elasticgraph-proto gem with stable proto field-number mappings #1056 on top of the new serializer extension points

What

add a new elasticgraph-proto_ingestion extension gem that generates schema.proto and proto_field_numbers.yaml
keep schema.proto on public GraphQL field names while storing private name_in_index overrides in the sidecar mapping file
emit the generated schema.proto artifact preamble with protobuf // comments so it remains valid proto syntax
preserve field numbers across public renames via field.renamed_from and update repo wiring/docs/tests
pin enum value numbers in an enums section of proto_field_numbers.yaml, so removing or reordering enum values never renumbers the rest (removed values keep their numbers reserved; 0 remains the generated *_UNSPECIFIED value)
validate proto package names (each dot-separated segment must be a valid protobuf identifier) and reject non-integer field/value numbers instead of letting Integer() silently truncate floats
resolve proto field types via type_ref.with_reverted_override so built-in scalars renamed with type_name_overrides keep working

This PR is stacked on #1294, which restores the schema-evolution declarations (field.renamed_from et al) to core elasticgraph-schema_definition. Referencing external proto enum types (proto_external_types) is split out into the stacked follow-up PR #1286.

Notes for review

elasticgraph-proto_ingestion is fully independent of elasticgraph-json_ingestion — no runtime or development dependency. Its specs activate only the proto extension. Rename metadata comes from the core field.renamed_from declaration (see Restore schema-evolution declarations to elasticgraph-schema_definition #1294), which proto generation reads directly from typed State members — no respond_to? probing. When no renames are declared, field numbers are simply assigned fresh (covered by a dedicated spec).
Natural follow-ups now that the declarations are core: consume type.deleted_field to emit protobuf reserved field numbers (wire-safety), and type.renamed_from to migrate a message's number block across type renames.
Enum value numbers are now wire-stable via the same sidecar mechanism as field numbers (this was previously flagged here as an open design question; it's resolved in this revision).

Verification

script/quick_build (the one spec failure on this machine — cli_new_spec under bundler 4 — reproduces identically on main and is environmental)
script/type_check, script/lint, script/spellcheck
script/run_gem_specs elasticgraph-proto_ingestion (100% line + branch coverage)
remaining run_misc_checks steps that CI previously never reached: config artifacts, dependency diagrams, CI yaml, licenses, README snippets, rake site:validate

References

Moves `field.renamed_from`, `type.renamed_from`, `type.deleted_field`, and `schema.deleted_type` — along with the state registries and `DeprecatedElement` record they populate — from `elasticgraph-json_ingestion` back into the core schema definition gem, partially reverting the placement chosen in #1259. These declarations record serializer-neutral facts about how a schema has evolved, and we now have a second consumer: `elasticgraph-proto_ingestion` (#1080) needs rename metadata to keep protobuf field numbers stable across renames (with `reserved` field numbers for deletions as a natural follow-up). Keeping the markers inside one serializer forced other consumers to duck-type against state that may or may not be present; with the declarations in core, every consumer reads typed `State` members directly. This also addresses the concern raised in review of #1259 about `elasticgraph-json_ingestion` bolting 9 fields onto `::ElasticGraph::SchemaDefinition::State`: its `StateExtension` now adds only the 5 genuinely JSON-specific fields, and the four deprecation registries are ordinary typed `State` members that need no `State & StateExtension` casts. All JSON-specific behavior — schema version enforcement, merge reporting, pruning, and the warnings that flag no-longer-needed declarations — stays in `elasticgraph-json_ingestion`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Fills in the `elasticgraph-proto_ingestion` gem with the Protocol Buffers schema artifact generation logic. Running `schema_artifacts:dump` emits: - `schema.proto` — the generated Protobuf schema for the indexed types - `proto_field_numbers.yaml` — a sidecar that reserves field numbers and enum value numbers so they stay wire-stable as the schema evolves (including across field renames, and reserving removed values so numbers are not reused) Capabilities: - Maps built-in ElasticGraph scalars to proto types, with `proto_field` to configure custom scalars. - Generates messages for object/interface/union types and enums (with a zero-valued `*_UNSPECIFIED` entry), escaping proto reserved words and wrapping nested lists so the output stays valid. - `proto_enum_mappings` reuses enum values already maintained elsewhere. The generator emits `proto3` by default and can emit `proto2` via `syntax: :proto2` (which labels every field `optional`/`repeated`). Arbitrary file-level headers (e.g. `option` declarations) can be injected verbatim via `headers:`, so language-specific options can be set without baking any particular convention into the gem.

myronmarston

Not done reviewing but wanted to submit what I have so far.

myronmarston · 2026-07-03T18:55:59Z

+        # @dynamic proto_schema_syntax, proto_schema_syntax=
+        # @dynamic proto_schema_headers, proto_schema_headers=
+        attr_accessor :proto_schema_package_name, :proto_enums_by_graphql_enum,
+          :proto_field_number_mappings, :proto_schema_syntax, :proto_schema_headers


Similar to #1281 can we move these all to a separate object and then have a single proto_state field on schema_def_api.state?

myronmarston · 2026-07-07T17:51:03Z

+  schema.scalar_type "Money" do |t|
+    t.mapping type: "long"
+    t.json_schema type: "integer"
+    t.proto_field type: "int64"


It feels odd for this to be called proto_field--we're configuring a type, not a field so why does the API say "field"? And it's not mapping_field/json_schema_field for the other options.

WDYT about calling this protobuf? e.g. t.protobuf type: "int64"?

myronmarston · 2026-07-07T17:51:24Z

+ElasticGraph.define_schema do |schema|
+  schema.proto_enum_mappings(
+    SalesEg::ProtoEnumMappings::PROTO_ENUMS_BY_GRAPHQL_ENUM
+  ) if defined?(SalesEg::ProtoEnumMappings)


Not sure you want to name a block-internal project here...

myronmarston · 2026-07-07T17:55:58Z

+```
+
+When a mapping exists for an enum, `elasticgraph-proto_ingestion` uses the mapped proto enum(s)
+as the source of enum values (respecting `exclusions`, `expected_extras`, and `name_transform`).


EG today doesn't have any concept of proto enum mappings, and doesn't have any logic to support exclusions, expected_extras or name_transform. Instead of importing an existing data structure designed for tests within Block internal code, can we invert this?

Design and implement whatever features make sense for the elasticgraph-proto_ingestion gem as a stand-alone, open source library.

Update the internal testing support inside Block to leverage what elasticgraph-proto_ingestion provides.

For example, instead of it being provided as a global data structure, maybe we can do something like:

schema.enum_type "Currency" do |t| t.external_proto_enum "v1.money.Currency", # ... plus other options as needed like `exclusions`, etc. end

With that said, I don't yet understand how this gem uses this mapping...I haven't gotten that far.

myronmarston · 2026-07-07T21:06:00Z

+RSpec.configure do |config|
+  config.when_first_matching_example_defined(:proto_schema) do
+    require "support/proto_schema_support"
+  end


You can also think about applying this tag automatically to specs under elasticgraph-proto_ingestion/spec, using this technique:

elasticgraph/elasticgraph-json_ingestion/spec/spec_helper.rb

Lines 16 to 18 in 4cdc065

config.define_derived_metadata(absolute_file_path: %r{/elasticgraph-json_ingestion/}) do |meta|

meta[:json_ingestion_schema_definition] = true

end

myronmarston · 2026-07-07T21:09:53Z

+| `Boolean`         | `bool`     |
+| `Cursor`          | `string`   |
+| `Date`            | `string`   |
+| `DateTime`        | `string`   |


Should we use https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp for this?

myronmarston · 2026-07-07T21:11:15Z

+|-------------------|------------|
+| `Boolean`         | `bool`     |
+| `Cursor`          | `string`   |
+| `Date`            | `string`   |


One potential concern is that string is a much wider type than Date. Something like 0.0001% of valid strings are valid ISO8601 Date strings.

Have you given thought to how invalid values will get handled? With JSON schema we could specify validations for this kind of thing.

myronmarston · 2026-07-07T21:13:13Z

+
+Additionally:
+- List types become `repeated` fields.
+- Nested list types generate wrapper messages so the output remains valid `proto3`.


I'm not following this.

schema.object_type "Team" do |t| t.field "players", "[Player!]!" do |f| f.mapping type: "nested" end t.field "seasons", "[Season!]!" do |f| f.mapping type: "object" end end

Is this saying that a wrapper message type will be generated for the players field but not the seasons field? Why would that be? And what would the proto look like?

jwils force-pushed the joshuaw/protobuf-schema-generation branch from 2e0e4a1 to 53ea492 Compare March 22, 2026 22:54

jwils force-pushed the joshauw/pluggable-ingestion-serializers branch from a3cfa5c to 9e6f39a Compare March 22, 2026 22:57

jwils force-pushed the joshuaw/protobuf-schema-generation branch 3 times, most recently from 4325d51 to fd8dd47 Compare March 23, 2026 04:10

jwils force-pushed the joshauw/pluggable-ingestion-serializers branch from 9e6f39a to c408a50 Compare March 23, 2026 04:10

jwils force-pushed the joshuaw/protobuf-schema-generation branch from fd8dd47 to bdce7fb Compare March 26, 2026 14:32

jwils force-pushed the joshauw/pluggable-ingestion-serializers branch from c408a50 to 5717334 Compare March 26, 2026 14:32

jwils force-pushed the joshauw/pluggable-ingestion-serializers branch 12 times, most recently from 0607029 to 40815e1 Compare April 4, 2026 21:53

jwils force-pushed the joshuaw/protobuf-schema-generation branch 6 times, most recently from 6591c29 to 751db75 Compare April 5, 2026 15:31

jwils force-pushed the joshauw/pluggable-ingestion-serializers branch 4 times, most recently from b2e8d34 to 6969f84 Compare May 6, 2026 15:42

jwils force-pushed the joshuaw/protobuf-schema-generation branch 3 times, most recently from 070849e to ce625bc Compare June 16, 2026 23:27

jwils mentioned this pull request Jun 20, 2026

Reference external proto enum types #1265

Closed

jwils force-pushed the joshuaw/protobuf-schema-generation branch 2 times, most recently from e9902f1 to f069179 Compare June 29, 2026 19:47

jwils marked this pull request as ready for review June 29, 2026 20:20

jwils requested review from BrianSigafoos-SQ, bsorbo, ellisandrews-toast, jwondrusch, marcdaniels-toast, myronmarston and rossroberts-toast as code owners June 29, 2026 20:20

jwils force-pushed the joshuaw/protobuf-schema-generation branch 2 times, most recently from 5efeb79 to 7d428d5 Compare July 1, 2026 20:55

jwils mentioned this pull request Jul 1, 2026

Reference external proto enum types #1286

Open

myronmarston reviewed Jul 2, 2026

View reviewed changes

Comment thread elasticgraph-protobuf/elasticgraph-protobuf.gemspec Outdated

jwils force-pushed the joshuaw/protobuf-schema-generation branch from 7d428d5 to 9acfed5 Compare July 4, 2026 01:38

jwils changed the title ~~Add elasticgraph-protobuf schema artifacts~~ Add elasticgraph-proto_ingestion schema artifacts Jul 4, 2026

jwils force-pushed the joshuaw/protobuf-schema-generation branch from 9acfed5 to 4fe8938 Compare July 4, 2026 12:59

jwils mentioned this pull request Jul 5, 2026

Restore schema-evolution declarations to elasticgraph-schema_definition #1294

Merged

jwils force-pushed the joshuaw/protobuf-schema-generation branch from 4fe8938 to 474788d Compare July 5, 2026 03:49

jwils changed the base branch from main to joshuaw/schema-evolution-declarations July 5, 2026 03:49

jwils force-pushed the joshuaw/protobuf-schema-generation branch from 474788d to bcc30c1 Compare July 5, 2026 15:39

jwils force-pushed the joshuaw/schema-evolution-declarations branch from 98f82ce to 683be9a Compare July 7, 2026 14:45

jwils mentioned this pull request Jul 7, 2026

Scaffold the elasticgraph-proto_ingestion gem #1297

Merged

myronmarston requested changes Jul 7, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add elasticgraph-proto_ingestion schema artifacts#1080

Add elasticgraph-proto_ingestion schema artifacts#1080
jwils wants to merge 1 commit into
mainfrom
joshuaw/protobuf-schema-generation

jwils commented Mar 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

myronmarston left a comment

Uh oh!

myronmarston Jul 3, 2026

Uh oh!

myronmarston Jul 7, 2026

Uh oh!

myronmarston Jul 7, 2026

Uh oh!

myronmarston Jul 7, 2026

Uh oh!

myronmarston Jul 7, 2026

Uh oh!

myronmarston Jul 7, 2026

Uh oh!

myronmarston Jul 7, 2026

Uh oh!

myronmarston Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	config.define_derived_metadata(absolute_file_path: %r{/elasticgraph-json_ingestion/}) do \|meta\|
	meta[:json_ingestion_schema_definition] = true
	end

Uh oh!

Conversation

jwils commented Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Notes for review

Verification

References

Uh oh!

Uh oh!

myronmarston left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jwils commented Mar 22, 2026 •

edited

Loading